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Abstract 

The  Office  of  Management  and  Budget  (OMB)  has  tasked  Federal  agencies  to 
develop  a  Data  Center  Consolidation  Plan.  Effective  planning  requires  a  repeatable 
method  to  effectively  and  efficiently  size  Air  Force  Base-level  data  centers.  Review  of 
commercial  literature  on  data  center  design  found  emphasis  in  power  efficiency,  thermal 
modeling  and  cooling,  and  network  speed  and  availability.  The  topic  of  sizing  data  center 
processing  capacity  seems  undeveloped.  This  thesis  provides  a  better,  pedigreed  solution 
to  the  data  center  sizing  problem.  By  analogy,  Erlang's  formulae  for  the  probability  of 
blocking  and  queuing  should  be  applicable  to  cumulative  CPU  utilization  in  a  data  center. 
Using  survey  data  collected  by  38th  Engineering  Squadron,  a  simulation  is  built  and 
correlation  between  the  observed  survey  measurements  and  simulation  measurements, 
and  the  Erlang,  Gamma,  and  Gaussian-Normal  distributions  is  found. 

For  a  sample  dataset  of  70  servers  over  14  hours  of  observation  and  a  supposed 
0.99999  requirement  for  traffic  to  be  passed  or  otherwise  unimpeded,  Erlang  distribution 
predicts  10  CPU  cores  are  required,  Gamma  distribution  predicts  10  CPU  cores  are 
required,  Gaussian-Normal  distribution  predicts  9  CPU  cores  are  required,  Erlang  B 
formulae  predicts  14  CPU  cores  are  required,  and  Erlang  C  formulae  predicts  15  CPU 
cores  are  required. 
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PARAMETRIC  ESTIMATION  OF  LOAD  FOR  AIR  FORCE  DATA  CENTERS 


I.  Introduction 


General  Issue 

From  the  1950’s  through  the  1970’s,  the  mainframe  data  center  was  the  only 
effective  means  of  computing.  Starting  in  the  1980’s  and  until  recently,  the  cost  of 
individual  computing  has  continued  to  drop.  An  overabundance  of  isolated  functional  and 
program  managed  data  centers  emerged.  Recently,  there  has  been  a  trend  towards 
consolidation  of  these  data  centers  to  gain  economies  in  staffing,  power,  environmental 
control,  reliability,  and  computing  power.  This  is  due  in  large  part  to  advances  in  high¬ 
speed  networking  technology  and  processor  virtualization,  which  allow  for  sharing  of 
processing  across  a  pool  of  hardware  computing  resources.  Sizing  this  pool  of  computing 
resources  remains  a  challenge  and  is  the  focus  of  this  thesis. 

Federal  mandates  through  the  Federal  Data  Center  Consolidation  Initiative 
(FDCCI)  require  a  75%  reduction  in  data  centers  across  all  federal  departments  by  FY 15. 
Specific  to  the  Department  of  Defense  (DoD),  a  40%  reduction  is  expected,  which 
equates  to  reducing  from  772  data  centers  to  428.  DoD  Core  Data  Center  (CDC) 
initiatives  have  a  target  objective  date  of  FY18.  The  discrepancy  between  the  FDCCI 
and  CDC  timelines  is  best  explained  by  the  complexity  and  cost  involved  in  data  center 
consolidation  [1], 

The  Office  of  Management  and  Budget  (OMB)  has  tasked  Federal  agencies  to 
develop  a  Data  Center  Consolidation  (DCC)  Plan  in  support  of  FDCCI  [2];  a  Presidential 
Directive  memo  was  later  issued  reinforcing  this  task  [3].  To  accomplish  this,  the 
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Department  of  Defense  is  virtualizing  all  servers  with  few  exceptions.  The  preferred 


approach  is  using  “cloud”  technologies,  which  is  a  “model  for  enabling  ubiquitous, 
convenient,  on-demand  network  access  to  a  shared  pool  of  configurable  computing 
resources”  [4].  The  issue  is  the  cloud  needed  to  house  all  these  virtual  servers  has  not  yet 
been  built,  designed,  or  sized. 

Problem  Statement 

One  problem  that  arises  in  data  center  consolidation  is  the  sizing  of  a  cloud's 
virtual  environment  -  effectively,  determining  the  number  of  central  processing  units 
(CPUs).  Sizing  is  accomplished  by  determining  the  load,  which  are  time  requests  from 
virtual  machines  for  CPUs.  The  duration  of  these  requests  is  a  random  variable,  related  to 
the  myriad  of  enterprise  applications  being  used.  Load  is  then  derived  from  the 
probability  distribution  for  the  number  of  CPU  resources  occupied  simultaneously. 

Using  this  probability  distribution,  the  optimal  size  for  a  data  center  virtual  environment 
can  be  found  that  minimizes  waste  (idle  CPUs)  while  avoiding  shortages  (no  free  CPUs 
when  requested). 

One  organization  that  is  providing  base  data  center  consolidation  support  is  the 
Air  Force  Space  Command’s  (AFSPC)  38th  Engineering  Squadron,  Tinker  AFB,  OK. 
When  tasked  with  sizing  new  data  centers,  a  group  there  found  there  was  no  published 
formulae  or  models  that  could  be  used  to  meet  the  task.  Initially  the  team  looked  at 
sizing  new  requirements  based  on  existing  average  CPU  utilization  and  scaling  that 
against  60  percent  of  capacity  of  the  cumulative  environment.  The  38th  design  team 
established  60  percent  as  a  reasonable  buffer  against  peaks  in  traffic  on  a  basis  of 
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experience  with  data  center  applications.  They  believed  this  would  provide  a  sufficient 
margin  to  account  for  any  transient  peaks  in  processing  requirements.  This  thesis 
provides  a  better,  pedigreed  solution  to  the  data  center  sizing  problem. 

Research  Focus  and  Investigative  Questions 

Spurred  by  the  major  initiatives  of  the  Department  of  Defense  to  consolidate  data 
centers  in  support  of  the  Federal  Data  Center  Consolidation  Initiative  legislated  by 
Congress,  this  thesis  answers  the  following: 

How  should  the  Air  Force  size  physical  processing  of  a  proposed  data  center? 

It  is  important  that  the  new  data  centers,  the  new  cloud,  be  designed  and  sized  to  support 
the  existing  applications  that  are  already  fielded.  It  is  also  necessary  to  be  able  to  project 
and  compare  costs  of  varying  cloud  implementations,  such  as  Infrastructure  as  a  Service 
(IaaS).  Infrastructure  as  a  Service  would  assume  that  DoD  would  outsource  the 
operation,  ownership  and  maintenance  of  all  computing  infrastructure  and  equipment  of 
the  data  center.  Effectively,  DoD  would  pay  a  service  provider  on  a  per-use  basis. 

An  analogy  is  hypothesized  between  the  Erlang  distribution  describing  the  load  of 
human  callers  in  a  telephony  system  and  of  virtual  machines  (hosts)  requesting  CPU  time 
from  a  Hypervisor.  The  Erlang  distribution  is  a  continuous  probability  distribution 
related  to  other  parametric  exponential  and  Gamma  distributions.  While  used  originally 
by  A.K.  Erlang  to  estimate  the  number  of  phone  calls  made  to  a  telephone  switch,  it  has 
general  applicability  to  many  traffic  engineering  and  queuing  problems.  This  thesis  will 
validate  the  use  of  the  Erlang  distributions  by  applying  the  same  general  methodology 
used  to  historically  size  telephony  systems  to  size  data  center  virtual  environments.  To 
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answer  the  overall  research  question,  a  few  investigative  questions  must  first  be  posed 
and  answered. 

1.  What  are  the  prevalent  metrics  for  computer  processing  emphasized  by  current 

practice,  or  found  in  the  academic  body  of  knowledge? 

2.  How  well  does  an  Erlang  distribution  approximate  data  center  CPU  load? 

3.  How  should  processing  in  data  centers  or  IaaS  projects  be  sized? 

Methodology 

To  attempt  to  answer  these  investigative  questions  a  review  of  the  technology 
involved  is  conducted  followed  by  statistical  analysis  of  measurements  taken  by  a  38th 
CEIG  survey  team  as  well  as  analysis  of  a  simulation  based  upon  measurements  taken 
during  that  survey.  Correlation  between  the  observed  survey  measurements,  as  well  as 
simulation  measurements,  and  the  Erlang,  Gamma,  and  Gaussian-Normal  distributions 
will  be  calculated  to  test  the  validity  of  the  proposed  sizing  solution. 

Assumptions/Limitations 

Limitations  influencing  this  study  include  the  cost  and  complexity  of  testing  on  a 
production  network,  the  critical  nature  of  the  enterprise  data  center  systems,  and  the 
limits  of  our  ability  to  measure  load  and  generate  perfectly  realistic  traffic.  This  thesis 
will  assume  that  Windows®  Performance  Monitor  can  provide  accurate  measurements  of 
load,  the  open  source  tool  LookBusy  can  generate  realistic  traffic,  and  that  these  results 
from  a  prototype  lab  running  Windows®  Hyper-V  are  sufficiently  typical  for  all 
hypervisors  [5].  This  thesis  will  also  assume  that  the  survey  data,  upon  which  analysis  is 
based,  is  typical  of  an  Air  Force  data  center. 
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Implications 

Federal  mandates  through  the  Federal  Data  Center  Consolidation  Initiative 
(FDCCI)  require  a  75%  reduction  in  federal  data  centers  by  FY 15,  with  DoD  expecting  a 
40%  reduction.  The  discrepancy  between  the  FDCCI  and  CDC  timelines  is  best 
explained  by  the  complexity  and  cost  involved  in  data  center  consolidation.  The  Office  of 
Management  and  Budget  (OMB)  has  tasked  Federal  agencies  to  develop  a  Data  Center 
Consolidation  (DCC)  Plan  in  support  of  FDCCI.  This  thesis  will  provide  a  repeatable 
method  to  effectively  and  efficiently  size  base-level  data  centers. 

Preview 

In  the  next  chapter  background  information  on  data  centers,  virtualization,  metrics 
for  data  centers,  and  some  tools  for  statistical  analysis  will  be  introduced.  In  subsequent 
chapters,  a  methodology  for  statistical  analysis  of  survey  results  and  for  simulation  of  the 
survey  data  in  a  virtual  environment,  then  results  from  one  such  survey  and  analysis  of 
those  results  as  well  as  analysis  of  simulation  results  based  upon  the  survey. 
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II.  Literature  Review 


Chapter  Overview 

The  purpose  of  this  chapter  is  to  overview  technologies,  metrics,  and  statistical 
analysis  tools  to  provide  a  background  for  the  thesis  question. 

Technology 

History  of  Technology 

The  electronic  computer  has  evolved  over  the  years.  The  industry  started  with 
mainframe  computers  for  laboratories  and  large  firms  in  the  1940s.  In  the  1960s  the 
integrated  circuit  led  to  the  development  of  the  minicomputer  and  later  the 
microprocessor  enabled  the  personal  computer.  In  the  1969s  the  ARPANET  was 
established  beginning  the  era  of  the  networked  computer  [6].  It  was  in  the  1970s  that 
virtualization  began  in  earnest  with  the  development  of  time- share  computing  on 
mainframe  computers.  Virtualization  is  defined  as  a  technology  which  “enables  several 
operating  systems  and  applications  to  run  on  one  physical  server  or  ‘host.’  Each  self- 
contained  virtual  machine  (VM)  is  isolated  from  the  others,  and  uses  as  much  of  the 
host’s  computing  resources  as  it  requires.  [7]” 

Virtualization  Technology 

As  time-share  computing  was  further  developed  modem  virtualization  technology 
emerged.  It  is  implemented  using  a  supervisory  program  referred  to  as  a  hypervisor 
virtual  machine  manager.  The  hypervisor  abstracts  the  hardware  and  presents  common 
interfaces  to  virtual  machines.  By  sharing  common  storage  it  becomes  possible  for  the 
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virtual  machines  to  almost  instantly  be  migrated,  over  the  network,  between  physical 
computers  [8]. 


Figure  1.  Hardware  Abstraction  by  the  Hypervisor 

Migration  is  used  to  balance  load  across  hosts  and  to  prevent  loss  of  service  when 
hardware  goes  into  maintenance  or  fails.  This  creates  a  common  pool  of  resources  across 
a  common  network  of  computers.  Policies  for  management  of  resources  in  this  common 
pool  are  addressed  Grit  and  Wood  in  "Virtual  machine  hosting  for  networked  clusters: 
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Building  the  foundations  for  "autonomic"  orchestration,"  and  "Black-box  and  gray-box 
strategies  for  virtual  machine  migration,"  [9]  [10].  Microsoft’s  Performance  and 
Resource  Optimization  (PRO)  feature  for  Hyper-V  Virtual  Machine  Manager  (VMM) 
allows  administrators  to  configure  target  utilization  for  host  hardware  in  virtual 
environments.  In  VMM  2008  the  default  target  CPU  utilization  is  90  percent  [11].  While 
some  details  of  implementation  differ  between  hypervisors,  and  there  are  performance 
differences  between  hypervisors,  these  features  are  common  across  all  modern 
hypervisors  [5].  The  cumulative  effect  of  these  features  underpinning  virtualization  is  that 
a  small  set  of  hardware  can  run  what  used  to  require  a  large  set  of  hardware  while  at  the 
same  time  offering  higher  redundancy  than  before.  Below,  figures  2  and  4  help  illustrate 
these  architectural  advantages  of  virtualization. 

Data  Center  Technology 

For  the  purposes  of  this  paper,  the  central  technological  aspect  of  the  data  center 
is  the  ability  to  process  and  store  data.  This  capability  is  often  expressed  in  terms  of 
Higher  Performance  Computer  (HPC)  systems.  Hussain  and  Malik  identify  three  types  of 
HPC  architecture:  Cluster,  Grid,  Cloud.  The  HPC  cluster  is  a  group  of  computers  with 
redundant  interconnections  that  form  a  highly  available  system  [12].  The  cluster  is 
centrally  managed  and  interconnected  across  a  LAN  environment.  One  example  would  be 
a  website  that  is  load-balanced  across  multiple  web  servers  in  a  data  center.  Frequently 
the  load  balancing  is  accomplished  by  sharing  an  IP  address  across  the  cluster.  The  load 
balancing  function  is  sometimes  performed  by  the  cluster  itself. 
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Figure  2.  Simple  HPC  Cluster  Architecture 

The  HPC  Grid  is  a  group  of  computers  that  use  the  Internet  to  spread  calculations 
out  across  low  cost  commodity  components.  A  grid  does  not  exist  in  a  single  data  center 
but  instead  in  across  multiple  data  centers  and  frequently  homes,  classrooms,  and  office 
floors  where  it  can  use  spare  compute  to  complete  calculations  for  the  grid.  The 
distributed  nature  of  the  grid  limits  the  workload  it  can  take  on.  Typically  grid  computing 
is  used  to  handle  non-interactive  workloads  that  can  be  broken  into  self-contained  chunks 
to  be  processed  by  grid  members.  The  same  chunk  may  be  sent  to  multiple  grid  members 
for  result  verification  or  to  mitigate  against  the  loss  of  a  grid  member. 
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Job  Flow 


Figure  3.  Simple  HPC  Grid  Architecture 

HPC  cloud  computing  is  an  amalgamation  of  grid  and  cluster  computing.  The 
cloud  can  exist  in  a  single  data  center  or  across  data  centers,  under  a  single  administrative 
domain  or  across  many,  as  a  private  commodity  or  as  a  public  service. 
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Figure  4.  Simple  HPC  Cloud  Architecture 

The  HPC  Cloud  architecture  is  shown  in  Figure  4.  Users  are  presented  with 
persistent  service  while  the  workload  is  shared  across  physical  data  center  resources.  The 
system  VMM  controls  scheduling  of  resources  on  each  system.  The  single  most  unique 
feature  of  the  cloud  computing  model  is  that  it  is  entirely  virtualized.  Where  computers 
participating  in  a  HPC  cluster  or  HPC  grid  can  be  virtualized  they  can  also  run  on  bare 
metal.  The  self-service  and  elasticity  characteristics  of  cloud  computing  requires  a 
common  pool  of  resources  to  exist  in  the  data  center  that  can  be  provisioned  on  request. 

Metrics 

Traditional  Data  Center  Metrics  -  Power,  Ping,  and  Pipe 

Review  of  the  literature  finds  remarkably  little  by  the  way  of  similar  or  alternative 

solutions  to  the  problem  of  sizing  a  data  center's  compute  environment.  Instead  there  is  a 
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great  emphasis  by  recent  authors  on  the  "power,  ping,  and  pipe"  [13]  associated  with  data 
center  design.  Other  work  describes  the  goals  of  "power  and  pipe"  in  terms  of  high  power 
usage  effectiveness,  PUE,  as  achieved  through  efficient  power  distribution  and  cooling 
systems.  Methods  for  achieving  "Ping,"  referring  to  the  need  for  a  responsive  and 
resilient  internetwork  in  the  data  center,  have  been  well  discussed  in  other  work  [14]  [15] 
[i6]  mi 

Load  Testing  Metrics 

There  is  an  existing  body  of  knowledge  on  the  use  of  software  load  generators  and 
the  analysis  and  identification  of  important  performance  counters.  Methods  vary  from 
emulation  of  production  networks,  stress  testing  of  individual  web  applications,  to 
arbitrary  load  generation  in  commercial  cloud  environments  such  as  the  Amazon  Elastic 
Cloud.  [18]  [19]  [20]  [21]  [22], 

Data  Center  Metrics 

In  order  to  study  data  center  capacity  and  utilization  it  is  necessary  to  establish 
metrics  and  collection  methods.  In  studying  a  1500  node  HPC  cluster  [23]  identified 
SNMP  counters,  Sampled  Flow,  and  Deep  Packet  Inspection  methods  to  measure  traffic 
patterns  and  performance.  These  are  network  focused  metrics.  SNMP  is  capable  of 
measuring  CPU  information  but  it  is  not  implemented  in  USAF  Windows  Server 
deployments.  Instead,  performance  counters  and  WMI  are  available  to  provide  reliable 
information  about  CPU  utilization.  Metrics  are  measured  over  the  interval  between  two 
measurement  instants  [24] . 
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Tools  for  Statistical  Analysis 

This  section  will  develop  and  introduce  concepts  of  statistical  analysis  by  way  of 
descriptive  statistics  including  terms  involved  in  calculations  and  distributions  used  by 
inferential  statistics.  The  purpose  of  descriptive  statistics  is  to  describe  and  summarize 
the  characteristics  of  a  sample.  To  accomplish  that,  measures  of  mean,  median,  standard 
deviation  and  variance  are  used.  These  measures  can  then  be  applied  to  distributions  to 
infer  properties  of  the  underlying  system.  To  describe  how  well  a  statistical  distribution 
fits  a  sample  dataset  the  correlation  coefficient  can  be  calculated  by  equation  1. 


Eq.  1 


where: 


Correl(X,Y)  =  the  correlation  coefficient  of  sets  X  and  Y 
x  =  Sample  of  X 
x  =  Mean  of  X 
y  =  Sample  of  Y 
y  =  Mean  of  Y 


The  Central  Limit  Theorem  is  a  theory  that  states,  convolution  of  a  sufficiently 
large  number  of  independent  random  variables  will  be  approximately  normally 
distributed.  This  distribution  is  also  referred  to  as  the  Gaussian-Normal  distribution  and 
shown  as  equation  2. 
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P(x)~- 


1 


e-(x-rf/2„> 

<J\/2l r  Eq.  2 

where: 

P(X)  =  Probability  Density  Function  of  Gaussian-Normal  Distribution 
x  =  Sample  value 
p  =  Mean 

a  =  Standard  Deviation 

The  Gaussian-Normal  distribution  is  one  of  the  simplest  random  function  used  for 
time-series  modeling  allowing  direct  calculation  from  just  mean  and  standard  deviation 
[25], 

Erlang’s  Formula 

This  section  will  develop  and  introduce  Erlang’s  formulas.  The  Erlang  function  is 
an  equation  initially  published  as  a  solution  to  telephony  problems  in  Elektroteknikeren 
Vol  13  (1917)  by  Agner  Krarup  Erlang  [26].  It  is  founded  in  the  theory  of  "statistical 
equilibrium"  by  which  ,  for  a  very  large  number  of  calls  the  individual  characteristics  of 
each  call  does  not  affect  the  group  characteristic  as  an  increase  in  one  call  duration  will 
be  balanced  by  a  decrease  in  some  other  call  duration.  Erlang  supposes  the  probability  of 
finding  all  telephone  lines  engaged  (a  blocking  condition  as  described  by  B  )  can  be 
approximated  by  the  total  number  of  lines  and  the  average  number  of  calls  per  time  unit 
(traffic  intensity)  [27].  These  dimensionless  units  of  traffic  intensity  have  come  to  be 
known  as  Erlangs.  The  blocking  probability,  Py,  is  also  known  as  the  Grade  of  Service. 
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where: 


Pb  =  Probability  a  request  for  service  is  blocked 
A  =  Normalized  load  or  mean  traffic  intensity 
N  =  Number  telephone  lines 


As  an  alternative  to  blocking  excess  traffic,  the  Erlang  C  formula  describes  the 
probability  that  a  call  is  placed  on  hold  and  waits  for  service  in  a  queue.  To  simplify  the 
mathematics  of  it,  it  is  assumed  that  callers  will  wait  indefinitely. 


Eq.  4 


where: 


Pw  =  Probability  a  call  has  to  wait  and  is  buffered 

A  =  Normalized  load  or  mean  traffic  intensity  =  call  frequency  x  call  duration 
N  =  Number  of  telephone  lines 


Under  certain  conditions  Pw  can  be  calculated  from  P/„  N,  and  A  without  an 
additional  round  of  summation.  Where  N  >  A  then 


Eq.  5 


where: 


Pw  =  Probability  a  call  has  to  wait  and  is  buffered 
Pt  =  Probability  a  request  for  service  is  blocked 
A  =  Normalized  load  or  mean  traffic  intensity 
N  =  Number  of  telephone  lines 


Erlang's  formula  for  solving  certain  problems  in  telephony  has  been  applied  to 
computer  networks  before.  Chromy  and  Baronak  [28]  applied  Erlang  distributions  to 
ATM  and  IP  network  traffic.  They  found  success  for  Erlang  B  in  the  case  of  ATM 
networks  and  Erlang  C  for  IP  networks,  noting  that  ATM  and  IP  handle  delay  differently. 
In  their  work,  the  Erlang  equations  were  not  modified  but  the  parameters  were. 
Specifically,  A  was  used  to  describe  Link  utilization  and  N  to  describe  Bandwidth. 

Table  1 :  Chromy  and  Kavacky's  common  parameters  of  synchronous  and 
asynchronous  networks 


Synchronous  Network 

Asynchronous  Network 

B  [%] 

Lost  calls  ratio 

B  [%] 

Loss  rate 

C 

Probability  of  waiting  for  service 

C 

Probability  of  delay 

A 

[Erl] 

Total  offered  traffic 

A  [%] 

Link  utilization 

N 

Number  of  channels  (links) 

N 

[Mbit/s] 

Bandwidth 

Bonald  and  Thomas  focused  on  IP  traffic  and  similarly  found  the  probability  that 
internet  traffic  (IP  traffic) ,  which  should  reduce  flow  rate  to  avoid  buffering,  is  bounded 
by  the  Erlang  C  formula  [29].  Bonald  and  Thomas,  similar  to  Chromy  and  Kavacky, 
identified  an  explicit  performance  relationship  involving  only  link  capacity  (N)  and 
expected  demand  (A). 

Erlang’s  Distribution 

This  section  will  introduce  the  Erlang  Distribution  and  the  Gamma  Distribution. 
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where: 


f(x;a,  f!)  =  Probability  Density  Function  of  Gamma  Distribution 
x  =  Sample  Value 
a  =  p~  /  a'  =  Distribution  Shape 
ft  =  a2  /  [i  =  Distribution  Rate 


The  Erlang  Distribution,  equation  7,  is  a  special  case  of  the  Gamma  Distribution, 
equation  6,  where  the  shape  parameter  a  is  a  positive  integer.  This  allows  the 
denominator  of  the  distribution  to  be  calculated  by  factorial  rather  than  by  the  Gamma 
function. 


Eq.  7 


where: 


f(x;a,  fi)  =  Probability  Density  Function  of  Erlang  Distribution 
v  =  Sample  Value 

a  =  p2  /  a2  =  Distribution  Shape,  a  positive  integer 
=  a2  /  p  =  Distribution  Rate 


The  NIST  published  Engineering  Statistics  Handbook  cites  the  Erlang 
Distribution  as  being  frequently  used  in  queuing  theory  applications  [25]. 

Summary 

The  idea  of  sharing  a  common  pool  of  computing  resources  is  not  a  new  one.  Old 
time-share  methods  have  been  replaced  by  virtualization  technology  which  is  able  to 
abstract  and  present  tasks  to  generic  hardware  in  a  data  center.  The  HPC  data  center  as  a 


cloud  offers  users  real-time  service,  with  resources  on  demand.  To  answer  the 
investigative  questions,  quantitative  metrics  are  needed.  Review  of  commercial  literature 
finds  emphasis  in  power  efficiency,  thermal  modeling  and  cooling,  and  network  speed 
and  availability.  The  topic  of  sizing  data  center  processing  capacity  seems  undeveloped. 
By  analogy  Erlang’s  formulae  seems  adaptable  to  the  problem,  however,  of  all  of  the 
reviewed  literature  only  one  reference  to  the  applicability  of  Erlang’s  methodology  to 
CPUs  could  be  found  [30].  Other  statistical  tools  from  descriptive  and  inductive 
statistical  analysis  can  be  used  to  further  analyze. 
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III.  Methodology 


Chapter  Overview 

As  discussed  earlier,  there  are  significant  similarities  between  the  telephony 
problems  of  1917  and  the  modern  virtualized  computing  environment.  The  greatest 
possibility  for  error  lay  in  the  analogy  of  call  frequency  and  duration  to  CPU  utilization. 
To  validate  the  adaptation  of  the  Erlang  formula  to  problems  of  the  modern  data  center  a 
medium  sized  virtual  environment  was  built  containing  several  virtualized  application 
servers  and  numerous  load  generating  "client"  virtual  machines.  Performance  monitors 
were  used  to  track  average  CPU  utilization  from  the  perspective  of  the  Guest  operating 
system,  as  well  as  the  System/Processor  Queue  Length.  Processor  Queue  Length  is 
proportional  to  the  amount  of  time  threads  were  awaiting  physical  compute  resources  and 
is  typically  less  than  12  [31].  This  emulated  data  will  be  processed  to  fit  parameters  to 
for  the  Erlang  formula  and  distribution,  as  well  as  the  Gamma  and  Gaussian-Normal 
distribution  using  correlation  coefficients. 

Erlang  Formulae  Modified  For  CPU  Loading 

Since  originally  the  Erlang  formulae  were  intended  for  sizing  telephone  circuits, 
the  definitions  for  some  variables  must  be  modified  to  fit  with  a  virtual  data  center 
environment  rather  than  a  telephony  environment.  Based  on  work  by  Chromy  and 
Kavacky  the  definitions  of  some  variables  involved  in  Erlang’s  formulae  should  be 
modified  using  Table  2. 
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Table  2  :  Common  parameters  of  synchronous  Telephony  and  Virtual  Data  Center 


Telephony 

Virtual  Data  Center 

Pb  [%] 

Lost  calls  ratio 

Pb  [%] 

Loss  rate 

Pw 

Probability  of  waiting  for  service 

Pc 

Probability  of  delay 

A  [Erl] 

Total  offered  traffic 

A  [%] 

Mean  CPU  utilization 

N 

Number  of  channels  (links) 

N 

Number  of  logical 

CPU  (cores) 

Applying  the  modified  parameters  from  table  2  to  equation  3  and  4  results  in 
equations  8  and  9  below. 


Eq.  8 


where: 


Pb  =  Probability  a  request  for  service  is  blocked 
A  =  %  Mean  CPU  Utilization 

N  =  Number  of  logical  CPU  available  to  the  hypervisor 


Eq.  9 


where: 


Pc  =  Probability  a  request  for  service  has  to  wait  and  is  buffered 
A  =  %  Mean  CPU  Utilization 

N  =  Number  of  logical  CPU  available  to  the  hypervisor 


Equations  associated  with  distributions  (Eq.  2,  6,  7)  are  also  modified  such  that  x 

2  2  2 

=  N  and  p  =  A  therefore  a  =  A"  /  a  and  /?  =  a  /  A.  The  CDF  of  the  Gaussian-Normal, 
Gamma  and  Erlang  distributions  is  produced  by  equation  10. 


Eq.  10 
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where: 


f(x)  =  Cumulative  Distribution  Function 
P(x)  =  Probability  Density  Function 

In  practice,  all  of  these  functions  are  calculated  by  Excel. 


Survey  Data  Collection 

When  tasked  with  sizing  new  data  centers  as  part  of  a  data  center  consolidation 
effort,  a  group  at  38th  Engineering  Squadron,  Tinker  AFB,  OK  sent  engineering  teams  to 
survey  the  existing  Air  Force  data  center  infrastructure  at  prospective  sites.  Over  the 
course  of  several  days  NIPRNET  connected  systems  were  identified,  the  system  owners 
identified  and  interviewed,  and  performance  information  was  collected  using  Windows 
Performance  Monitor  from  the  production  application  servers.  Twenty-seven  CPU 
Utilization  samples  were  taken  in  30  minute  intervals  over  the  course  of  14  hours  for 
each  of  70  identified  application  servers  beginning  at  10  am.  Metrics  are  measured  over 
the  interval  between  two  measurement  instants  [24] .  An  example  of  the  survey  data  is 
shown  in  table  3.  To  protect  the  OPSEC  of  the  servers  their  names  have  been  replaced 
with  numbers.  This  CPU  Utilization  information,  included  in  Appendix  B,  is  used  to 
generate  scripts  for  simulation  of  the  consolidated  data  center  and  for  direct  statistical 
analysis  in  Chapter  4. 

Table  3  :  Example  of  Survey  Data 


Server 

%utilization 

Time 

Server 

%utilization 

Time 

Server 

%utilization 

Time 

005v 

0 

0 

006v 

0 

0 

007v 

1 

0 

005v 

0 

30 

006v 

0 

30 

007v 

28 

30 

005v 

0 

60 

006v 

0 

60 

007v 

26 

60 

005v 

0 

90 

006v 

0 

90 

007v 

24 

90 

005v 

1 

120 

006v 

0 

120 

007v 

2 

120 
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Experimental  Equipment  Setup 

A  Dell  PowerEdge  R720  with  eight  10,000  RPM,  900  GB  SAS  harddrives, 
twenty-two  1600MHz  16GB  DIMMs,  and  two  2.70GHz  Xeon  E5-2697  CPUs  is  used  as 
the  host  system.  Each  Xeon  CPU  provides  the  host  12  cores.  After  Hyper-threading,  the 
host  operating  system  is  presented  48  logical  processors.  There  is  a  LI  cache  of  1.5MB,  a 
L2  cache  of  6MB,  and  a  L3  cache  of  60MB.  Harddrives  2-6  are  combined  into  one 
3353GB  logical  drive  with  Windows  2012  soft  RAID-5.  Harddrive  1  contains  data 
unrelated  to  the  project.  Harddrive  7  is  used  to  store  installation  files.  Harddrive  8  is  the 
system  drive  of  the  host  operating  system.  The  host  system  also  has  a  3000GB  Fusion-io 
drive2  installed  and  recognized  by  the  host  operating  system  as  disk  0,  it  is  used  as  part  of 
a  tiered  storage  pool  to  overcome  disk  IO  limitations  found  during  early  testing. 


Figure  5.  Experimental  Equipment  Setup  and  Configuration 
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Software  Configuration 

The  software  configuration,  and  key  aspects  of  its  interaction  with  hardware,  is 
depicted  in  figure  5.  Windows  Server  2012R2  is  used  as  the  host  operating  system.  The 
Hyper-V  hypervisor  is  enabled  allowing  virtualization  of  additional  guest  operating 
systems  (OS)  and  applications.  The  RAID-5  partition  is  labeled  “D.”  One  virtual  switch 
named  “NICO  Bridged”  is  created  and  associated  with  all  guest  OS.  The  host  operating 
system  is  able  to  use  this  virtual  switch  to  communicate  with  guest  OS. 

The  guest  operating  systems  are  Windows  Server  2012R2.  One  virtual  machine 
was  created  with  8GB  of  RAM,  2  virtual  CPU,  and  a  virtual  60GB  harddrive  on  the  D 
drive.  After  installation  of  Windows  Server  2012R2,  Cygwin,  Lookbusy,  and  cpu-load- 
generator.py  are  loaded  and  a  scheduled  task  is  created  to  start  the  simulation.  The 
template  virtual  machine  is  shutdown  and  cloned  to  simulate  an  Air  Force  Data  Center. 

Simulation  Generation 

To  create  the  data  center  workload  trace  twenty- seven  samples,  one  per  half  hour 
over  a  fourteen  hour  period,  were  taken  from  70  systems  in  the  Air  Force  Data  Center. 
These  were  then  enumerated  into  trace  files  for  use  by  cpu-load-generator.py.  The 
simulation  was  configured  to  change  state  every  9  seconds  with  sampling  every  3 
seconds  to  satisfy  Shannon's  Theorem.  Three  simulation  sets  were  created: 

•  Replay  of  survey  data 

•  Random  sampling  of  survey  data  set  per  application 

•  Random  sampling  of  survey  data  set  per  unit  time 
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The  first  simulation  is  intended  to  show  validity  of  the  simulation  in  a  consolidated 
virtual  environment  and  is  expected  to  closely  match  the  survey  data  which  was  taken 
from  a  distributed  physical  environment.  The  second  simulation  is  intended  to  simulate 
the  data  center  with  an  assumption  that  load  between  applications  is  not  dependant  on 
time.  The  third  simulation  is  intended  to  simulation  the  data  center  with  an  assumption 
that  load  between  applications  is  dependent  on  time.  A  python  script  was  developed  for 
the  purpose  of  generating  the  second  and  third  simulation  sets  and  is  included  in 
Appendix  A. 

Workload  Generation 

The  cpu-load-generator.py  script  was  developed  by  Dr.  Beloglazov  as  an  open 
source  tool  to  generate  CPU  load  according  to  a  configuration  profile  or  workload  trace. 
It  uses  Devin  Carraway’s  open  source  tool  Lookbusy  to  make  an  arbitrary  number  of 
CPUs  arbitrarily  busy.  While  Dr.  Beloglazov  used  a  web  service  to  assign  and  trigger 
load  profiles,  in  this  simulation  task  scheduler  is  used  with  load  profiles  assigned  by 
powershell  script. 

Each  clone  is  configured  with  a  CPU  load  profile  based  on  measurements  taken 
by  the  38ES  survey  team.  This  allows  the  virtual  environment  to  either  replay  real 
activity  seen  in  a  physical  Air  Force  Data  Center  or  run  simulation  scenarios  described  in 
the  previous  section. 
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Data  Collection 


The  Windows  Performance  Monitoring  tool,  Perfmon,  was  used  to  collect  data. 
Perfmon  allows  for  the  recording  of  hundreds  of  different  counters.  In  the  simulation 
phase,  this  study  used  several  counters  including  [32]: 


Table  4  :  Windows  Performance  Monitoring  Metrics 


Metric 

Description 

Example 

<Hyper-V  Hypervisor  Virtual 
Processor\CPU  Wait  Time  Per 
Dispatch> 

The  average  time  (in  nanoseconds)  spent 
waiting  for  a  virtual  processor  to  be 
dispatched  onto  a  logical  processor 

12699.25 

<  Hyper-V  Hypervisor  Virtual 
Processor(_Total)\%  Total  Run 

Time  > 

Shows  the  time  the  cpu  is  not  idle  across  the 
Hyper-V  environment 

5.243332 

<System\Processor  Queue  Length> 

Shows  the  number  of  threads  waiting  to  be 
serviced 

0 

<  Hyper-V  Hypervisor  Virtual 
Processor(_Total)\%  Hypervisor 

Run  Time> 

Slice  of  the  %  Total  Run  Time  used  by  the 
Virtual  Machine  Manager 

0.052604 

<  Hyper-V  Hypervisor  Virtual 
Processor(_Total)\%  Guest  Run 
Time> 

Slice  of  the  %  Total  Run  Time  used  by  guest 
virtual  machines 

5.190729 

<System\File  Write 

Operations/sec> 

Shows  the  combined  rate,  in  incidents  per 
second,  of  file  system  write  requests  to  all 
devices  on  the  computer 

61.32717 

<System\File  Read  Operations/sec> 

Shows  the  combined  rate,  in  incidents  per 
second,  of  file  system  read  requests  to  all 
devices  on  the  computer 

60.66057 

<System\File  Control 
Operations/sec> 

Shows  the  combined  rate,  in  incidents  per 
second,  of  file  system  operations  that  were 
neither  read  nor  write  operations. 

271.6394 

During  data  collection  in  the  production  environment,  a  PowerShell  script  was  used  to 
sample  the  Perfmon  value  for  <System\%  Processor  Time>  [33]. 
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Analysis  Methodology 

There  are  five  sections  of  data  to  be  analyzed.  Raw  survey  data,  cumulative 
survey  data,  cumulative  replay  simulation  data,  cumulative  application-based  simulation 
data,  and  cumulative  time-based  simulation  data.  The  raw  survey  data  consists  of  CPU 
utilization  measurements  taken  by  the  38  ES  survey  team  on  a  per  application  server 
basis.  Cumulative  survey  data  consists  of  CPU  utilization  measurements  summed  across 
all  70  application  servers  in  the  data  center  for  each  sample  in  time.  The  sum  of  all 
samples  at  t=0  is  the  cumulative  CPU  utilization  at  t=0,  the  sum  of  all  samples  at  t=30  is 
the  cumulative  CPU  utilization  at  t=30,  and  so  on.  All  three  Cumulative  simulation  data 
sets  consist  of  total  CPU  utilization  measurements  from  the  hypervisor  supporting  the 
simulation.  The  cumulative  simulation  data  sets  offer  a  view  into  resource  utilization  in  a 
fully  virtual  environment. 
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Figure  6.  Cumulative  CPU  Utilization  of  Survey  data,  Stacked  Line  Plot 
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°  Time  (  Minutes  from  survey  start) 


Figure  7.  Surface  Plot  of  Survey  Data 

The  analysis  methodology  is  based  on  chapter  2  findings  for  Erlang’s  formulae 
and  more  generally  for  statistical  analysis  from  the  Engineering  Statistics  Handbook.  For 
each  data  set,  descriptive  statistical  values  for  minimum,  maximum,  mean  and  standard 
deviation  are  calculated.  A  histogram  is  plotted  then  compared  by  correlation  with 
Erlang’s  formulae  then  with  the  Erlang,  Gamma  and  Gaussian-Normal  distributions.  As 
noted  above,  the  Erlang  distribution  will  be  calculated  by  rounding  the  a  parameter  to  the 
nearest  integer. 
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IV.  Analysis  and  Results 


Chapter  Overview 

In  this  part  the  results  obtained  by  calculation  from  measurements  obtained  during 
survey,  as  well  as  during  simulation  and  compare  with  Erlang,  Gamma,  and  Gaussian- 
Normal  distributions  are  presented. 

Descriptive  Statistical  Analysis 

Appendix  B  includes  an  anonymized  version  of  Windows  Performance  Monitor 
<System\%  Processor  Time>  data  collected  during  the  survey  of  an  Air  Force  Data 
Center.  Server  names  were  replaced  with  numbers  from  00 lv  to  070 v. 

Since  the  Performance  Monitor  results  were  collected  into  a  csv  format,  Excel 
was  used  to  calculate  descriptive  statistics  that  summarize  the  raw  survey  data.  Table  5 
compares  these  statistics.  All  values,  except  for  sample  count,  are  expressed  in  terms  of 
physical  CPU  cores. 

A  median  value  of  zero  for  Raw  Survey  Data  indicates  that  at  least  50%  of 
samples  reported  no  server  activity,  emphasizing  possible  efficiencies  to  be  found  in 
virtualization  and  consolidation  by  showing  that  over  50%  of  the  time  data  center  assets 
are  simply  waiting  for  work. 
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Table  5  :  Comparison  of  Descriptive  Statistics  for  Each  Data  Set 


Survey  Data 

Cumulative  Survey  Data 

Cumulative  Replay  Simulation  Data 

Minimum 

0 

Minimum 

2.81 

Minimum 

2.59 

Maximum 

1 

Maximum 

5.83 

Maximum 

14.1 

Mean 

0.0629 

Mean 

4.24 

Mean 

4.60 

Median 

0 

Median 

4.23 

Median 

4.63 

Samples 

1838 

Samples 

27 

Samples 

2850 

Standard 

Deviation 

0.136 

Standard  Deviation 

0.937 

Standard  Deviation 

0.814 

Cumulative  App-based  Simulation 
Data 

Cumulative  Time-based  Simulation 

Data 

Minimum 

2.16 

Minimum 

2.19 

Maximum 

9.3 

Maximum 

8.4 

Mean 

4.59 

Mean 

4.56 

Median 

4.55 

Median 

4.52 

Samples 

5401 

Samples 

5401 

Standard  Deviation 

0.651 

Standard  Deviation 

0.810 

For  most  cumulative  data  sets,  the  minimum,  mean,  and  standard  deviation  of 
CPU  utilization  are  similar.  The  cumulative  app-based  simulation  data  set  shows  a 
smaller  standard  deviation  when  compared  with  the  other  simulation  scenarios  and  results 
in  a  larger  a  value  and  smaller  P  value  per  definitions  for  equations  6  and  7  as  well  as 
slightly  lower  predictions  for  hardware  requirements. 

Analysis  of  Fit  of  Erlang’s  Formulae 

The  second  part  of  analysis  uses  equations  8  and  9  to  attempt  to  size  a  data  center. 
The  descriptive  parameters  of  mean  CPU  utilization,  A,  is  used  together  with  various 
numbers  of  CPU  cores  available  for  traffic,  N,  then  compared  with  the  inverse 
cumulative  distribution  of  the  observed  datasets. 
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Erlang  C 


%  CPU  Utilization 


Figure  8.  Comparison  of  Erlang  B  and  C  with  Survey  Data 
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Figure  9.  Comparison  of  Erlang  B  and  C  with  Cumulative  Survey  Data 


30 


Figure  10.  Comparison  of  Erlang  B  and  C  with  Cumulative  Replay  Simulation  Data 
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Figure  11.  Comparison  of  Erlang  B  and  C  with  Cumulative  App-based  Simulation 
Data 
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Figure  12.  Comparison  of  Erlang  B  and  C  with  Cumulative  Time-based  Simulation 
Data 


The  Erlang  B  and  Erlang  C  formulae  do  not  appear  to  be  a  good  fit  in  figures  8  - 
12.  Calculation  of  correlation  coefficients  in  table  6  shows  some  degree  of  correlation. 

Table  6  :  Comparison  of  Erlang  Formulae  Correlation  Coefficients  for  Each  Data 
Set 


Erlang  B 

Erlang  C 

Cumulative  Survey  Data 

0.950820839 

0.937472778 

Cumulative  Replay  Simulation  Data 

0.941291397 

0.921250068 

Cumulative  App-based  Simulation  Data 

0.931956494 

0.911316961 

Cumulative  Time-based  Simulation  Data 

0.941643605 

0.922631406 

Raw  Survey  Data 

0.877987087 

0.791652648 

Other  inferential  statistical  methods  discussed  in  Chapters  2  and  3  are  next  pursued. 
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Inferential  Statistical  Analysis 

The  third  part  of  analysis  uses  descriptive  parameters  calculated  in  the  previous 
section  as  inputs  to  known  statistical  distributions  and  will  use  correlation  coefficients  to 
find  best  fitting  distributions.  Beginning  with  the  raw  survey  data,  visualized  in  Figure 
13,  it  is  shown  that  the  Gamma  distribution  is  a  very  close  match  to  the  observed  data.  It 
is  also  apparent  that  the  Erlang  distribution  is  not  a  good  fit  here.  This  is  caused  by  the 
distribution  shape  parameter  for  the  Erlang  distribution,  a,  which  in  this  case  was  less 
than  0.5  to  begin  with,  being  rounded  to  the  nearest  positive  integer  of  1.  As  per 
equations  6  and  7,  the  distribution  shape  is  a  product  of  p2  /  a2,  or  A2  /  a2  when  translated 
by  the  modified  parameters  table,  table  2. 
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Figure  13.  Raw  Survey  Data  -  Comparison  of  Observed,  Gamma  (a  =  0.215 ,  ft  = 
0.292),  Gaussian-Normal  (p  =  0.063,  a  =  0.136)  and  Erlang  (a  =  1 ,  ft  =  0.292) 

Distributions 

The  Erlang,  Gamma,  and  Gaussian-Normal  distributions  are  compared  with  the 
observed  replay  simulation  data  in  Figures  14,  15,  and  16.  The  same  distributions  are 
compared  with  the  application  based  simulation  data  in  Figures  17,  18,  19  and  again  with 
the  time  based  simulation  data  in  Figures  20,  21,  and  22.  Each  comparison  was  split  out 
into  its  own  figure  to  improve  readability.  Each  of  the  three  distributions  match  so 
closely,  however,  it  is  still  difficult  to  differentiate  between  the  two  curves. 
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Figure  14.  Cumulative  Replay  Simulation  Data  -  Observed  CDF  vs  Erlang 
Distribution  CDF  (a  =  32,p  =  0.14419) 
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Figure  15.  Cumulative  Replay  Simulation  Data  -  Observed  CDF  vs  Gamma 
Distribution  CDF  ( a  =  31.901,  p  =  0.14419) 
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Figure  16.  Cumulative  Replay  Simulation  Data  -  Observed  CDF  vs  Gaussian- 
Normal  Distribution  CDF  (p  =  4.5999,  o  =  0.81441) 
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Figure  17.  Cumulative  App-based  Simulation  Data  -  Observed  CDF  vs  Erlang 
Distribution  CDF  (a  =  50,  fi  =  0.09239 5) 
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Figure  18.  Cumulative  App-based  Simulation  Data  -  Observed  CDF  vs  Gamma 
Distribution  CDF  (a  =  49.664,  p  =  0.092395) 
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Figure  19.  Cumulative  App-based  Simulation  Data  -  Observed  CDF  vs  Gaussian- 
Normal  Distribution  CDF  (p  =  4.5888,  a  =  0.65114) 
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Figure  20.  Cumulative  Time-based  Simulation  Data  -  Observed  CDF  vs  Erlang 
Distribution  CDF  (a  =  32,  p  =  0.14377) 
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Figure  21.  Cumulative  Time-based  Simulation  Data  -  Observed  CDF  vs  Gamma 
Distribution  CDF  (a  =  31.727,  p  =  0.14377) 


38 


_Q 

ns 

J2 

O 

i— 

CL 

cm 

c 


1 

0.9 

0.8 

0.7 

0.6 

0.5 

0.4 

0.3 

0.2 

0.1 

0 


Observed  CDF 
Gaussian  CDF 


1  2  3  4  5  6  7  8  9  10 11 12  13  14 15  16 17 

Utilized  CPU  Cores 


Figure  22.  Cumulative  Time-based  Simulation  Data  -  Observed  CDF  vs  Gaussian- 
Normal  Distribution  CDF  (p  =  4.5229,  a  =  0.80984) 

In  order  to  overcome  the  limitations  of  the  graphs,  correlation  coefficients  are  shown  in 
table  7.  It  is  shown  that  for  cumulative  data  sets,  all  three  distributions  are  good  fits  and 
that  the  Gaussian-Normal  distribution  provides  the  best  fit.  For  the  raw  survey  data  it  is 
shown  that  the  Gamma  distribution  provides  the  best  fit. 

Table  7  :  Comparison  of  Distribution  Correlation  Coefficients  for  Each  Data  Set 


Erlang  CDF 

Gamma  CDF 

Gaussian 

CDF 

Cumulative  Survey  Data 

0.998495255 

0.999039326 

0.998980715 

Cumulative  Replay  Simulation  Data 

0.999155381 

0.999106357 

0.999349484 

Cumulative  App-based  Simulation  Data 

0.999935781 

0.999986389 

0.999953034 

Cumulative  Time-based  Simulation  Data 

0.999936780 

0.999991168 

0.999937383 

Raw  Survey  Data 

0.8272* 

0.99664592 

0.862088045 

*  Could  not  be  calculated  because  a  was  too  small  after  rounding 
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In  trying  to  understand  these  results,  it  is  important  to  recall  the  Central  Limit  Theorem. 


While  individual  application  workloads  do  not  appear  to  be  best  fit  by  a  Gaussian- 
Normal  distribution,  the  hypervisor  is  essentially  performing  convolution  of  the  various 
distributions  representing  each  individual  application  in  the  data  center.  As  such,  for  a 
sufficiently  large  number  of  applications  virtualized  in  a  data  center,  the  distribution  will 
tend  towards  Gaussian-normal. 


Contrasting  with  Sixty  Percent  Rule 

The  Sixty  Percent  Rule  discussed  in  Chapter  1,  and  used  by  the  38th  CEIG  team 
during  early  planning  efforts,  multiplies  mean  utilization  measured  during  the  survey  ( 
4.24  )  by  1.6  then  rounds  up  to  the  nearest  integer.  Table  8  compares. 

Table  8  :  Comparison  of  Estimated  CPU  Requirements 


CPU  Cores  Required 
(5  9s) 

CPU  Cores  Required 
(3  9s) 

Cumulative  Survey  Data  -  Gaussian 

9 

8 

Cumulative  Replay  Simulation  Data  - 
Gaussian 

9 

8 

Cumulative  App-based  Simulation  Data  - 
Gaussian 

8 

7 

Cumulative  Time-based  Simulation  Data  - 

Gaussian 

9 

8 

Cumulative  Survey  Data  -  Gamma 

10 

8 

Cumulative  Replay  Simulation  Data  -  Gamma 

9 

8 

Cumulative  App-based  Simulation  Data  - 
Gamma 

8 

7 

Cumulative  Time-based  Simulation  Data  - 

Gamma 

9 

8 

Cumulative  Survey  Data  -  Erlang 

10 

8 

Cumulative  Replay  Simulation  Data  -  Erlang 

9 

8 

Cumulative  App-based  Simulation  Data  - 
Erlang 

8 

7 

Cumulative  Time-based  Simulation  Data  - 
Erlang 

9 

8 
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Cumulative  Survey  Data  -  Erlang  B 

14 

13 

Cumulative  Replay  Simulation  Data  -  Erlang  B 

15 

13 

Cumulative  App-based  Simulation  Data  - 
Erlang  B 

15 

13 

Cumulative  Time-based  Simulation  Data  - 
Erlang  B 

15 

13 

Cumulative  Survey  Data  -  Erlang  C 

15 

13 

Cumulative  Replay  Simulation  Data  -  Erlang  C 

15 

14 

Cumulative  App-based  Simulation  Data  - 
Erlang  C 

15 

14 

Cumulative  Time-based  Simulation  Data  - 
Erlang  C 

15 

13 

Sixty  Percent  Rule 

7 

7 

Evidently,  the  Sixty  Percent  Rule  matches  up  with  Gaussian,  Erlang  and  Gamma 
distribution  predictions  for  a  data  center  with  3  nines  reliability  under  the  Application 
based  simulation  scenario.  Under  all  other  scenarios,  including  calculations  based  off  of 
the  survey  data  itself,  the  Sixty  Percent  Rule  under  estimates  the  hardware  requirement. 

Investigative  Questions  Answered 

1.  What  are  the  prevalent  metrics  for  computer  processing  emphasized  by 
current  practice,  or  found  in  the  academic  body  of  knowledge? 

In  research,  the  majority  of  the  academic  body  of  knowledge  centers 
around  optimization  of  data  center  (electrical)  power,  (network)  ping,  and 
(cooling)  pipes.  It  has  perhaps  been  the  prevalence  of  Moore’s  Law, 
popularly  thought  of  as  processor  power  doubling  every  18  months,  that 
has  left  the  topic  of  estimating  processing  requirements  in  neglect. 

2.  How  well  does  an  Erlang  distribution  approximate  data  center  CPU  load? 

A  quirk  of  the  Erlang  distribution  is  a  parameter  of  the  distribution  must  be  a 
positive  integer.  During  the  analysis  of  the  collected  data,  the  Gamma 
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distribution’s  CDF  (which  is  the  same  as  the  Erlang  distribution  sans  the  integer 
requirement)  was  used  and  found  to  strongly  correlate  to  both  the  simulation  and 
survey  datasets.  For  the  simulation  datasets,  with  over  2000  samples,  a  normal 
distribution  was  found  to  high  slightly  higher  correlation  coefficient  than  a 
gamma  distribution,  0.9993  compared  to  0.9991.  For  the  survey  data,  with  only 
27  samples,  the  reverse  was  true,  gamma  distribution  CDF  correlation  coefficient 
0.9990  compared  to  normal  distribution  CDF  of  0.9989.  This  is  not  a  significant 
difference  in  correlation.  In  cases  where  there  are  fewer  virtual  servers  in  the 
environment  it  should  be  expected  for  the  gamma  distribution  to  be  more  reliable 
whereas  in  environments  with  many  virtual  servers,  such  as  the  70  found  in  the 
38th  ES  survey  and  simulated  as  part  of  this  thesis,  the  Gaussian-Normal 
distribution  has  been  shown  to  be  reliable  as  well. 

3.  How  should  processing  in  data  centers  or  IaaS  projects  be  sized? 

Since  the  Erlang/Gamma  distribution  was  shown  to  be  a  good  fit  for  the  observed 
data,  IaaS  and  data  center  projects  may  use  Erlang’s  distribution,  Gamma  distribution,  or 
Gaussian-Normal  distribution  to  help  size  necessary  processing  capacity.  In  situations 
where  a  small  number  of  servers  are  being  virtualized,  the  Gamma  distribution  was  found 
to  be  the  most  accurate.  Where  a  large  number  of  servers  are  being  virtualized,  the 
Gaussian-Normal  distribution  should  be  used  to  minimize  waste  capacity. 
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V.  Conclusions  and  Recommendations 


Chapter  Overview 

In  this  chapter,  conclusions  and  recommendations  for  action  and  future  research 
are  presented. 

Conclusions  of  Research 

By  analogy,  Erlang’s  formulae  for  the  probability  of  blocking  and  queuing  were 
expected  to  be  applicable  to  cumulative  CPU  utilization  in  a  data  center.  One  assumption 
of  the  Erlang  formulae  is  that  the  traffic  essentially  follows  an  exponential  distribution. 
While  this  was  found  to  be  true  for  individual  applications,  it  was  not  so  once  the 
hypervisor  convoluted  dozens  of  virtual  applications  against  limited  physical  resources. 
The  research  shows  the  distribution  of  cumulative  CPU  utilization  across  a  large  number 
of  applications  (70  application  servers  in  the  survey  and  simulation)  can  be  described  by 
the  Erlang/Gamma  distribution  and  Normal  distribution.  While  it  appears  that  Erlang’s 
formulae,  with  minor  modification  discussed  in  chapter  3,  can  be  used  for  CPU  loads,  a 
normal  distribution  is  more  accurate. 

This  is  significant  to  Air  Force  Data  Center  Consolidation  efforts  because  it 
allows  planners  to  estimate  how  much  consolidation  can  take  place  using  existing 
resources  and  what  new  resources  will  be  required  to  reach  the  end  state.  Without  a 
method  for  parametric  estimation  of  load  in  Air  Force  datacenters,  planners  are  forced  to 
make  wild  guesses  at  the  requirement.  This  leads  to  either  over  purchasing  hardware  to 
avoid  the  risk  of  not  having  enough,  which  wastes  money,  or  under  purchase  hardware 
for  lack  of  justification,  which  reduces  reliability  and  increases  latency  of  Air  Force 
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Enterprise  applications  running  in  the  datacenter  like  Active  Directory  Services  and 
Microsoft  Exchange. 

Recommendations  for  Action 

Statistical  analysis,  taking  advantage  of  either  Gamma  or  Gaussian-Normal 
distribution,  should  be  used  to  test  if  processing  capacity  is  a  likely  cause  of  delay  when 
sizing  and  troubleshooting  virtual  environments.  As  a  practical  example,  if  an  availability 
of  5  nines  is  required  for  a  base  level  data  center  then  mean  cumulative  CPU  utilization 
plus  5  standard  deviations  calculates  the  processing  requirement  before  any  additional 
redundancy  requirements  are  introduced. 

Recommendations  for  Future  Research 

Future  research  should  include  analysis  of  changes  to  cumulative  CPU  utilization  induced 
by  hypervisor  co-scheduling  processes  and  methods  for  predicting  mean  CPU  utilization 
in  virtual  environments  based  on  measurements  of  the  same  application  on  a  physical 
host.  Future  research  might  also  include  investigation  of  the  use  of  Markov  chains  as 
statistical  models  to  better  predict  CPU  utilization. 
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Appendix  A:  Testbed  Setup  and  Configuration 

Cygwin  was  installed  to  allow  for  the  use  of  the  open  source  load  generator  LookBusy.  A 
python  script  originally  written  by  Anton  Beloglazov  was  used  to  control  the  LookBusy 
load  generator.  Cygwin  needed  the  Archive,  Python,  WEB->wget,  DEVEL->gcc,  and 
DEVEL->make  packages  included  for  successful  installation  and  operation. 


Survey  Dataset 

This  file  is  used  as  an  input  to  the  simulation-generator.py  script 


LB  Utilization.csv 


Simulation-generator.py 

#  Copyright  2015  Derek  Molle 

#  Usage  of  the  works  is  permitted  provided  that  this  instrument  is 
#retained  with  the  works,  so  that  any  entity  that  uses  the  works  is 
#notified  of  this  instrument.  Disclaimer:  The  works  are  without 
fwarranty . 

#This  script  uses  LB  Utilization.csv  to  generate  two  folders  of 
#simulation  files  for  use  by  cpu-load-generator . py :  asim  where  a 
#simulation  is  generated  using  application  behavior  as  basis,  tsim 
fwhere  a  simulation  is  generated  using  data  center  behavior  at  each 
tmoment  in  time  as  a  basis 
import  csv 
import  random 

fnumber  of  simulations  to  generate  for  both  application  and  time 
#methods 

numsimulations  =  100 

with  open (' LB  Utilization.csv',  ' rb ' )  as  surveydataf ile : 
csvdata  =  csv. DictReader (surveydataf ile) 

data  =  { x . strip (): [y]  for  x,y  in  csvdata . next (). items () } 

for  rows  in  csvdata: 

for  x,y  in  rows . items () : 

data [x . strip ( ) ] . append (y) 
print  len (data [ ' Time ' ] ) 

timedistributions  =  { x . strip ():[ ]  for  x  in  set (data [' Time ']) } 
appdistributions  =  { x . strip ():[ ]  for  x  in  set (data [' App ']) } 
for  t,  u,  a  in  zip (data [' Time '], data [' Utilization '], data [ 'App ']) : 
timedistributions [t . strip ( ) ] . append (u) 
appdistributions [a . strip ( ) ] . append (u) 

LBAppfiles  =  { x . strip ():[ ]  for  x  in  set (data [' App ']) } 
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LBTime files  =  {x. strip ():[]  for  x  in  set (data [ 1 App ' ] ) } 


for  x  in  LBAppfiles: 

LBAppf iles [x]  =  [random. choice (appdistributions [x] )  for 
in  xrange (numsimulations*27 ) ] 

for  in  xrange (numsimulations ) : 

for  t  in  set (data [' Time ']) : 
for  x  in  LBTimefiles: 

LBTimef iles [x] . append (random. choice (timedistributions [t] ) ) 

path  =  "asim\\" 

for  x  in  LBAppfiles: 

f ile=open (path+x+" . txt"  ,  ' w ' ) 
for  y  in  LBAppfiles [x] : 

file . write ( "%s\n"  %  y) 
path  =  "tsim\\" 
for  x  in  LBTimefiles: 

f ile=open (path+x+" . txt"  ,  ' w ' ) 
for  y  in  LBTimefiles [x] : 

f ile . write ( "%s\n"  %  y) 


cpu-load-generator.py 


#  Copyright  2012  Anton  Beloglazov 

# 

#  Licensed  under  the  Apache  License,  Version  2.0  (the  "License"); 

#  you  may  not  use  this  file  except  in  compliance  with  the  License. 

#  You  may  obtain  a  copy  of  the  License  at 

# 

#  http : // www . apache . org/ licenses /LICENSE-2 . 0 

# 

#  Unless  required  by  applicable  law  or  agreed  to  in  writing,  software 

#  distributed  under  the  License  is  distributed  on  an  "AS  IS"  BASIS, 

#  WITHOUT  WARRANTIES  OR  CONDITIONS  OF  ANY  KIND,  either  express  or 
implied . 

#  See  the  License  for  the  specific  language  governing  permissions  and 

#  limitations  under  the  License. 


"""  A  tool  for  generating  a  set  of  subsequent  CPU  utilization  levels. 

fl  II  II 


from  optparse  import  OptionParser ,  Option,  IndentedHelpFormatter 

import  os 

import  subprocess 

import  time 


def  process ( interval ,  utilization  list,  ncpus) : 
ncpus_str  =  str (ncpus) 
for  utilization  in  utilization^list : 

utilization  str  =  str (utilization) 

print  "\nSwitching  to  "  +  utilization  str  +  "%" 

p  =  subprocess . Popen ([' lookbusy '  , 

'--ncpus',  ncpus_str. 
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' --cpu-util ' ,  utilization_str] ) 

time . sleep ( interval ) 
p . terminate ( ) 


class  PosOptionParser (OptionParser ) : 

def  format  help (self,  formatter=None) : 
class  Positional (object) : 

def  _ init _ (self,  args) : 

self . option_groups  =  [] 
self . option^list  =  args 

positional  =  Positional ( self . positional ) 
formatter  =  IndentedHelpFormatter ( ) 
formatter .store  opt ion_st rings (positional ) 
output  =  ['\n',  formatter . format  heading (' Positional 
Arguments ' ) ] 

formatter . indent  ( ) 

pos_help  =  [ formatter . format  option (opt)  for  opt  in 
self . positional ] 

pos_help  =  [ line . replace ('--' ,  '')  for  line  in  pos_help] 

output  +=  pos_help 

return  OptionParser . format  help (self,  formatter)  + 

'  '  . j  oin (output) 

def  add_positional  argument (self ,  option): 
try : 

args  =  self . positional 
except  AttributeError : 
args  =  [] 

args . append (option) 
self . positional  =  args 

def  set_out (self ,  out) : 
self. out  =  out 


def  main ( ) : 

parser  =  PosOptionParser ( 

usage= ' Usage :  python  %prog  [options]  INTERVAL  SOURCE', 
description= ' Generates  a  set  of  subsequent  '  + 

'CPU  utilization  levels  read  from  a  file.  '  + 
'  '  + 
'Copyright  (C)  2012  Anton  Beloglazov.  '  + 
'Released  under  Apache  2.0  license.') 
parser . add_positional_argument ( 

Option (' --INTERVAL ' ,  action= ' store  true', 

help= ' interval  between  subsequent  CPU  '  + 
'utilization  levels  in  seconds')) 
parser . add_positional  argument ( 

Option ( ' --SOURCE ' ,  action= ' store_true '  , 

help=' source  file  containing  a  new  line  '  + 
'separated  list  of  CPU  utilization  levels  '  + 
'specified  as  numbers  in  the  [0,  100]  range')) 
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parser . add_option (' -n ' ,  ' --ncpus',  type='int',  dest= ' ncpus ' , 

def ault=l , 

help=' number  of  CPU  cores  to  utilize  [default 

1]  ') 

(options,  args)  =  parser . parse_args ( ) 
if  len (args)  ! =  2 : 

parser . error (' incorrect  number  of  arguments') 


try : 

interval  =  int(args[0]) 
except  ValueError: 

parser . error (' interval  must  be  an  integer  >=  O') 
if  interval  <=  0: 

parser . error (' interval  must  be  an  integer  >=  O') 
filename  =  args[l] 

if  not  os . access (filename,  os.R_OK): 

parser . error (' cannot  read  file:  '  +  filename) 

utilization  =  [] 
for  line  in  open (filename) : 
if  line . strip ( ) : 
try : 

n  =  float (line) 
if  n  <  0  or  n  >  100: 

raise  ValueError 
utilization. append (int (n) ) 
except  ValueError: 

parser . error (' the  source  file  must  only  '  + 
'contain  new  line  separated  '  + 
'numbers  in  the  [0,  100]  range') 


if  interval  <=  0: 

parser . error (' interval  must  be  an  integer  >=  O') 
process ( interval ,  utilization,  options . ncpus ) 


if  name  ==  '  main 
main  ( ) 
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lb.c  -  Lookbusy 


LB.c.docx 
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Appendix  B:  Survey  Information 
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Appendix  C  :  CDF  Tables 


Cumulative  Replay 


Simulation 

Data 

Cores  in 

Use 

Observed 

CDF 

Erlang  CDF 

Gamma 

CDF 

Gaussian 

CDF 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

2 

0 

0 

0 

0.0007 

3 

0.0112 

0.0135 

0.0142 

0.0247 

4 

0.2660 

0.2328 

0.2384 

0.2307 

5 

0.6400 

0.6981 

0.7041 

0.6884 

6 

0.9740 

0.9464 

0.9482 

0.9572 

7 

0.9996 

0.9952 

0.9954 

0.9984 

8 

0.9996 

0.9998 

0.9998 

1 

9 

0.9996 

1 

1 

1 

10 

0.9996 

1 

1 

1 

11 

0.9996 

1 

1 

1 

12 

0.9996 

1 

1 

1 

13 

0.9996 

1 

1 

1 

14 

0.9996 

1 

1 

1 

15 

1 

1 

1 

1 

16 

1 

1 

1 

1 
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Cumulative  App-based  Simulation 
Data 


Cores  in 

Use 

Observed  CDF 

Erlang 

CDF 

Gamma 

CDF 

Gaussian 

CDF 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

2 

0 

0 

0 

0.0000 

3 

0.0033 

0.0026 

0.0030 

0.0073 

4 

0.1766 

0.1718 

0.1845 

0.1829 

5 

0.7508 

0.7302 

0.7455 

0.7361 

6 

0.9791 

0.9761 

0.9785 

0.9849 

7 

0.9981 

0.9993 

0.9994 

0.9999 

8 

0.9993 

1.0000 

1.0000 

1 

9 

0.9998 

1 

1 

1 

10 

1.0000 

1 

1 

1 

11 

1.0000 

1 

1 

1 

12 

1.0000 

1 

1 

1 

13 

1.0000 

1 

1 

1 

14 

1.0000 

1 

1 

1 

15 

1 

1 

1 

1 

16 

1 

1 

1 

1 
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Cumulative  Time -based  Simulation 
Data 


Cores  in 

Use 

Observed 

CDF 

Erlang 

CDF 

Gamma 

CDF 

Gaussian 

CDF 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

2 

0 

0 

0 

0.0008 

3 

0.0176 

0.0140 

0.0160 

0.0269 

4 

0.2568 

0.2376 

0.2533 

0.2440 

5 

0.7139 

0.7040 

0.7203 

0.7059 

6 

0.9539 

0.9484 

0.9531 

0.9621 

7 

0.9967 

0.9955 

0.9960 

0.9987 

8 

0.9996 

0.9998 

0.9998 

1 

9 

1.0000 

1 

1 

1 

10 

1.0000 

1 

1 

1 

11 

1.0000 

1 

1 

1 

12 

1.0000 

1 

1 

1 

13 

1.0000 

1 

1 

1 

14 

1.0000 

1 

1 

1 

15 

1 

1 

1 

1 

16 

1 

1 

1 

1 
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Cumulative  Survey 
Data 


Cores  in  Use 

Observed 

CDF 

Erlang  CDF 

Gamma 

CDF 

Gaussian 

CDF 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

2 

0 

0.0023 

0.0016 

0.0085 

3 

0.0741 

0.0976 

0.0809 

0.0936 

4 

0.4444 

0.4667 

0.4278 

0.4007 

5 

0.7407 

0.8262 

0.8007 

0.7927 

6 

1 

0.9667 

0.9593 

0.9702 

7 

1 

0.9958 

0.9946 

0.9984 

8 

1 

0.9996 

0.9995 

1 

9 

1 

1 

1 

1 

10 

1 

1 

1 

1 

11 

1 

1 

1 

1 

12 

1 

1 

1 

1 

13 

1 

1 

1 

1 

14 

1 

1 

1 

1 

15 

1 

1 

1 

1 

16 

1 

1 

1 

1 
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